We describe an incentive system for distributed deep learning of foundational models where peers are rewarded for contributions. The incentive system, \textit{Gauntlet}, has been deployed on the bittensor blockchain and used to train a 1.2B LLM with completely permissionless contributions of pseudo-gradients: no control over the users that can register or their hardware. \textit{Gauntlet} can be applied to any synchronous distributed training scheme that relies on aggregating updates or pseudo-gradients. We rely on a two-stage mechanism for fast filtering of peer uptime, reliability, and synchronization, combined with the core component that estimates the loss before and after individual pseudo-gradient contributions. We utilized an OpenSkill rating system to track competitiveness of pseudo-gradient scores across time. Finally, we introduce a novel mechanism to ensure peers on the network perform unique computations. Our live 1.2B run, which has paid out real-valued tokens to participants based on the value of their contributions, yielded a competitive (on a per-iteration basis) 1.2B model that demonstrates the utility of our incentive system.

Sparse AutoEnocders (SAEs) have recently been employed as an unsupervised approach for understanding the inner workings of Large Language Models (LLMs). They reconstruct the model's activations with a sparse linear combination of interpretable features. However, training SAEs is computationally intensive, especially as models grow in size and complexity. To address this challenge, we propose a novel training strategy that reduces the number of trained SAEs from one per layer to one for a given group of contiguous layers. Our experimental results on Pythia 160M highlight a speedup of up to 6x without compromising the reconstruction quality and performance on downstream tasks. Therefore, layer clustering presents an efficient approach to train SAEs in modern LLMs.

Loss functions drive the optimization of machine learning algorithms. The choice of a loss function can have a significant impact on the training of a model, and how the model learns the data. Binary classification is one of the major pillars of machine learning problems, used in medical imaging to failure detection applications. The most commonly used surrogate loss functions for binary classification include the binary cross-entropy and the hinge loss functions, which form the focus of our study. In this paper, we provide an overview of a novel loss function, the Xtreme Margin loss function. Unlike the binary cross-entropy and the hinge loss functions, this loss function provides researchers and practitioners flexibility with their training process, from maximizing precision and AUC score to maximizing conditional accuracy for a particular class, through tunable hyperparameters $\lambda_1$ and $\lambda_2$, i.e., changing their values will alter the training of a model.

Deep Learning is a subfield of machine learning: a new take on learning representations from data that puts emphasis on learning successive layers of increasingly meaningful representations. How many layers contribute to a model of the data is called the depth of the model. The specification of what a layer does to its input data is stored in the layer's weights, which in essence are a bunch of numbers. In technical terms, we'd say that the transformation implemented by a layer is parameterized by its weights. Wights are also called the parameters of a layer.

Deep Learning is a subset of machine learning. Technically, machine learning looks through input data for valuable representations, making use of feedback signal as guidance. The "deep" in deep learning isn't a reference to any kind of deeper understanding attained by the approach; rather, it stands for the idea of consecutive layers of representations. The "depth" of the model is the indication of the number of layers subscribed to the model. Concurrently, shallow learning's (another approach of machine leaning) focal point is to learn representations from merely one or two layers.

In this post we covered a brief history of AI and how it evolved over the years through symbolic AI, ML and DL. We also tried to understand how AI, ML and DL are related/correlated with one other (figure 6). These terms often muddle a lot of people and are often used interchangeably. There is a lot of hype around DL models due to their ability to better deal with complex problems such as image classification, text processing etc. Additionally these models also remove the necessity to manually engineer features, amenable to the objective at hand. Therefore, in the next post we shall be delving into the details of DL models and try to understand its various components along with examples.

Classically, ML models trained with stochastic gradient descent (SGD) are designed to minimize the average loss per example and use a distribution of training examples that remains {\em static} in the course of training. Research in recent years demonstrated, empirically and theoretically, that significant acceleration is possible by methods that dynamically adjust the training distribution in the course of training so that training is more focused on examples with higher loss. We explore {\em loss-guided training} in a new domain of node embedding methods pioneered by {\sc DeepWalk}. These methods work with implicit and large set of positive training examples that are generated using random walks on the input graph and therefore are not amenable for typical example selection methods. We propose computationally efficient methods that allow for loss-guided training in this framework. Our empirical evaluation on a rich collection of datasets shows significant acceleration over the baseline static methods, both in terms of total training performed and overall computation.